What is Feature Selection

= Select a subset of all features for model training, usually in data preprocessing.

• Advantages
  • avoid the curse of dimensionality
  • improves the predictive performance and interpretability of models
    • shorten training time
    • prevent overfitting
    • reduce generalization error
• Domain knowledge is important!!!

Feature Selection methods

There are some of the techniques used for feature selection in data analysis

Intrinsic methods

• embedded methods or implicit methods: model already can handle it, do not require extra methods
• examples
  • tree-based models: a feature is not used in any split already means it is independent of the target variable
  • regularizaton models (Lasso regression Cost Functions#^ed4ab0, Ridge Regression Cost Functions#^aef45a): only keep features with non-zero coefficients

Filter methods

• select features that correlate well with target variables
• examples
  • univariate statistical analysis: select features with higher correlations with target variables
  • feature importance-based: select features with higher Interpretable Machine Learning#Feature Importance scores
    • via coefficient (linear regression, logistic regression) - but this may remove non-linear relationships
    • via impurity-based feature importances (tree-based models)

Wrapper methods

• iterative process that repeatedly add subjects of feature to the model
• examples
  • Sequential Feature Selection (SFS)
    • Forward SFS: start with 0 feature -> sequentially add feature
      1. select a significance level (SL, e.g. SL=0.05)
      2. fit all simple regression models. Select the one with the lowest P-value
      3. keep this variable and fit all possible models with one extra predictor added to the one(s) you already have.
      4. consider the predictor with the lowest P-value. If P < SL, go to step 3, otherwise go to FIN and keep the previous model
    • Backward SFS ( (Recursive Feature Elimination, fastest): start with all features-> sequentially remove feature
      1. select a significance level (SL, e.g. SL=0.05) 2. fit the full model with all possible predictors
      2. consider the predictor with the highest P-vales. If P > SL, go to step 4, otherwise go to FIN
      3. remove the predictor
      4. fit model without this variable
      5. return to step 3
  • Bidirectional Elimination (stepwise regression) = combination of Backward Elimination and Forward Selection
    1. select a significance level to enter and stay in the model (SL, e.g. SL=0.05)
    2. perform the next step of forward selection (new variables must have P < SL_enter to enter)
    3. perform all steps of backward elimination (old variables must have P < SL_enter to stay)
    4. repeat step 2 and 3, until no new variables can enter and no old variables can exit

Feature Selection Stability

= checks whether selected features remain important across different samples of the data.

Why it matters

• A feature selected once may be selected only because of one lucky train/test split.
• High-dimensional and correlated features can make feature lists unstable.
• Stable features are stronger candidates for interpretation or biomarker prioritization.
• For non-sparse models, a feature can appear important in one split but drop in rank in another split.

Common approaches

• Resampling
  • see Resampling-based Model Stability Checks#Common resampling schemes
• Stability measures:

Typical workflow for sparse models

1. Subsample the training data many times
2. Fit a sparse model each time
3. Record which features are selected each time
4. Calculate selection frequency for each feature
5. Rank features by how often they are selected

Typical workflow for non-sparse models

1. Resample the training data many times
2. Fit the model each time
3. Calculate feature importance each time, e.g. coefficients, permutation importance, tree-based importance, or SHAP values
4. Compare feature rankings or top-k important features across resamples
5. Check whether the same features remain consistently important